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(57) A multimedia information arranging apparatus 
that utilizes various feature values contained in multi- 
media information, carries out a flexible retrieval and 
classification and displays the information in an informa- 
tion set arrangement space is provided. 

An information set obtaining portion 10 obtains in- 
formation sets. An axis setting portion 20 assigns the 
feature value of the media information to an information 
set arrangement space axis so as to set the information 
set arrangement space. A feature value extracting por- 
tion 30 extracts the feature value from each of the media 
information of the obtained information set, and an in- 
formation set arranging portion 40 arranges the infor- 
mation sets in the information set arrangement space 
according to their feature values. An information dis- 
playing portion 50 displays the information set arrange- 
ment space in such a manner as to be seen from a pre- 
determined display viewpoint. By seeing the display re- 
sult, as necessary, the axis setting portion 20 resets the 
information set arrangement space axis using a different 
feature value, so that the information sets are rear- 
ranged and redisplayed in the information set arrange- 
ment space, and retrievals of different aspects are re- 
peated. 
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Description 

Technical Field 

[0001] The present invention relates to a multimedia 
information arranging apparatus that can arrange effi- 
ciently and flexibly a multimedia information group, in 
which not only text information but also image informa- 
tion and audio information are mixedly present as vari- 
ous media information. It also is possible to arrange a 
multimedia information group that is present accessibly 
on the World Wide Web (hereinafter, abbreviated as 
WWW) of the internet. 

Background Art 

[0002] Currently, there is a large amount of stored and 
accessible multimedia data including not only text infor- 
mation but also image information and audio informa- 
tion. In particular, accompanying the development of the 
internet, the amount of information on the WWW (World 
Wide Web) has been increasing steadily, so that there 
is a large amount of various kinds of stored and acces- 
sible multimedia data including the text information, the 
image information and the audio information. As a re- 
sult, it has become increasingly difficult to retrieve nec- 
essary information accurately from the WWW. 
[0003] Conventionally known information retrieval 
systems on the WWW and information arranging oper- 
ations therein are described in the following. 
[0004] First, a text key retrieval system is known. With 
respect to multimedia data to be a retrieval target, one 
or more pieces of text information representing their 
contents are assigned suitably. When retrieving the da- 
ta, a text such as a word is specified as a retrieval key, 
whereby the text key retrieval system retrieves the mul- 
timedia data to which text information that is the same 
as this retrieval key has been assigned. In the case 
where the media information to be a retrieval target is 
text information, since the contents of the text informa- 
tion are retrieved by a text key, such a text key retrieval 
generally is carried out in an appropriate manner. In this 
text key retrieval system, the information can be ar- 
ranged, for example, by sorting the data that have been 
hit in the retrieval and displaying a list thereof. 
[0005] Second, a full-text retrieval system is known. 
This retrieval system is particularly effective when mul- 
timedia data to be a retrieval target are pieces of text 
information. There are several suggested methods 
therefor. For example, as a preprocessing, all the words 
representing features of the contents of the full text are 
extracted automatically from the full text, and a retrieval 
key file is generated so that these words are assigned 
thereto. When retrieving the data, a text such as a word 
is specified as a retrieval key, thereby detecting the text 
information whose retrieval key file contains a text that 
is the same as this retrieval key. With this method, like 
the first retrieval method using the text as a key, media 



information other than the text information, for example, 
the image information, is difficult to retrieve appropriate- 
ly by using the text as a key, unless a suitable text key 
is assigned to each piece of the image information. 
s [0006] In this full-text retrieval system, the information 
also can be arranged, for example, by sorting the data 
that have been hit in the retrieval and displaying a list 
thereof, as in the text key retrieval system. 
[0007] Third, as a method for retrieving the image in- 
'0 formation, further known is an image retrieval method 
of image pattern matching by specifying a part of an im- 
age and using the partial image as a retrieval key. With 
this image retrieval method, the image information hav- 
ing the specified partial image can be retrieved. 
15 [0008] In this image retrieval system of image pattern 
matching, the information also can be arranged, for ex- 
ample, by sorting the data that have been hit in the re- 
trieval and displaying a list thereof, 
[0009] However, the conventional information retriev- 
al systems on the WWW do not involve an effective re- 
trieval method for the media information other than the 
text information, for example, the image information and 
the audio information. 

[0010] With the firstly-described conventional text key 
retrieval system, it is difficult to retrieve the image infor- 
mation and the audio information appropriately. In other 
words, even if a plurality of keywords are assigned to 
one piece of image information, it is still difficult to cany 
out an appropriate and flexible image retrieval by the 
keywords in accordance with a searcher's intention be- 
cause of the difficulty in representing the feature of the 
image accurately and flexibly with the text. Even when 
the retrieval is carried out indirectly using a text retrieval 
server or the like, the resultant information is not very 
useful. Although it is possible to collect a lot of images 
and display them sequentially, there has been a problem 
in that too many images would make it difficult to find an 
intended image. 

[0011] With the secondly-described conventional full- 
text retrieval system, it is difficult to retrieve the:image 
information and the audio information appropriately as 
in the first text key retrieval system. In other words, since 
the image information and the audio information origi- 
nally do not have text information, no text information 
extraction can be expected therefrom. 
[0012] With the thirdly-described conventional image 
retrieval system of image pattern matching, it is possible 
to retrieve the image information having a specified par- 
tial image. However, a searcher has to prepare and 
specify a partial image contained in the image that he/ 
she wants to retrieve. This makes it difficult to retrieve 
the image information because what kind of partial im- 
age is contained in the image that he/ she wants to re- 
trieve the most is not clear in some cases and a partial 
image used for the retrieval cannot always be prepared. 
Moreover, a searcher sometimes does not know clearly 
the image that he/ she wants to retrieve. In other words, 
there are some cases where a searcher can specify a 
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general content of the image that he/ she wants to re- 
trieve but cannot specify the image itself even partially. 
There also are cases where a searcher just wants to 
carry out a trial-and-error retrieval to find out any usable 
image indeterminately. In such cases, matching only 5 
partial images is not flexible enough and, therefore, in- 
sufficient. 

Disclosure of Invention 

[0013] It is an object of the present invention to pro- 
vide a multimedia information arranging apparatus that 
can retrieve multimedia information such as text infor- 
mation, image information and audio information effi- 
ciently and flexibly by utilizing various feature values 
contained in the multimedia information and arrange 
and display the retrieval result visually and understand- 
ably. In particular, it is an object of the present invention 
to provide a multimedia information arranging apparatus 
that can retrieve multimedia information on the WWW 
efficiently and flexibly and arrange this information. 
[0014] It is a further object of the present invention to 
provide a multimedia information arranging apparatus 
in which a searcher can narrow down desired multime- 
dia information flexibly based on the above-mentioned 
displayed retrieval result in an interactive manner and 
carry out one retrieval after another by a further aspect 
based on the retrieved multimedia information. 
[0015] In order to achieve the above-mentioned ob- 
jects, a multimedia information arranging apparatus of 
the present invention uses an information set as a 
processing unit. The information set is formed by group- 
ing together the pieces of media information that are re- 
lated to a same target, the media information being of a 
same kind and different kinds, from among a group of 
pieces of media information including image informa- 
tion, text information and audio information. Using the 
information set of the present invention as a processing 
unit as above is clearly different from a conventional 
multimedia processing. The conventional multimedia 
processing mainly refers to the following two process- 
ings. The conventional multimedia processing first 
means that one apparatus can deal with a plurality of 
media, which are image information, text information 
and audio information. In this case, although one appa- 
ratus can deal with a plurality of media, the processing 
unit itself is directed to each piece of media information. 
That is, each piece of the media information such as the 
image information, the text information and the audio in- 
formation is processed individually. The conventional 
multimedia processing secondly means that the 
processing unit itself is multimedia data into which a plu- 
rality of media have been integrated. In this case, the 
image information and the audio information are embed- 
ded or a link thereof is provided in the text information, 
for example. That is, the data themselves are integrally 
processed. On the other hand, the information set of the 
present invention formed by associating with each other 



related pieces of media information such as the image 
information, thelext information and the audio informa- 
tion, the media information being of the same kind and 
different kinds, is used as a processing unit and is dif- 
ferent from the case where each piece of media infor- 
mation is processed individually as in the firstly-de- 
scribed conventional multimedia information. Further- 
more, although data of plural pieces of media informa- 
tion are associated so as to be a set in the present in- 
vention, the data of the plural pieces of media them- 
selves are not integrally processed unlike the secondly- 
described conventional multimedia information, but 
rather each piece of the media information is maintained 
as they are collected and obtained. 
[0016] The multimedia information arranging appara- 
tus of the present invention includes an information set 
obtaining portion for obtaining pieces of media informa- 
tion in units of information sets described above, an axis 
setting portion for assigning an attribute of a feature val- 
ue extracted from the obtained pieces of media informa- 
tion contained in the information sets to an axis of a 
space in which a group of the information sets is ar- 
ranged and setting an information set arrangement 
space with one or more axes, a feature value extracting 
portion for extracting a component of the feature value 
from the pieces of media information in the information 
sets, an information set arranging portion for arranging 
the information sets in the information set arrangement 
space based on the attribute of the feature value of the 
pieces of media information contained in the information 
sets and the component of this feature value, and an 
information displaying portion for displaying pieces of 
media information corresponding to a viewpoint with re- 
spect to the information set arrangement space, from 
among the pieces of media information of the informa- 
tion sets arranged in the information set arrangement 
space. 

[0017] With the above configuration, it is possible to 
collect and obtain efficiently multimedia information 
such as text information, image information and audio 
information on the WWW as an information set, retrieve 
it efficiently and flexibly by utilizing various feature val- 
ues and display the retrieval result visually and under- 
standably in an information set arrangement space. 
[0018] Next, in the multimedia information arranging 
apparatus described above, in the axis setting portion, 
a plurality of the attributes of the feature values can be 
assigned in combination to one axis of the space or one 
attribute of the feature value can be assigned to a plu- 
rality of the axes. 

[0019] In the generation of the information set, the in- 
formation set obtaining portion includes an information 
collecting portion for collecting the pieces of media in- 
formation including the image information, the text infor- 
mation and the audio information, a relationship analyz- 
ing portion for analyzing a relationship between the col- 
lected pieces of media information and an information 
set generating portion for grouping and editing the re- 
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lated pieces pf media information, which is of the same 
kind or the different kinds, as the information sets, so 
that the group of the related pieces of media information 
may be formed into the information set Of course, the 
information sets that have been generated already may 
be read from a recording medium such as a CD-ROM 
by an information set input portion or collected from a 
network by an information set collecting portion. By 
grouping together the related pieces of media informa- 
tion into the information sets as described above, the 
text information and the audio information can be asso- 
ciated with the image information. Therefore, when car- 
rying out a retrieval using a feature value regarding the 
audio information and that regarding the text infomna- 
<ion, for example, the image information that is associ- 
ated with them also can be obtained at the same time. 
[0020] Also, the feature value to be used can be a 
DCT coefficient feature value with respect to the image 
information, a wavelet transform coefficient feature val- 
ue with respect to the image information or a HSI color 
histogram feature value with respect to the image infor- 
mation. It can be a feature value representing a pres- 
ence of a specific word with respect to the text informa- 
tion or a feature value of how many times a specific word 
is used with respect to the text information It can be a 
voice frequency feature value with respect to the audio 
information, an amplitude feature value with respect to 
the audio information or a time change feature value 
with respect to the audio information. 
[0021] Then, in the multimedia information arranging 
apparatus described above, it is preferable that the axis 
setting portion has an axis resetting function of resetting 
an assignment of the attribute of the feature value to 
each of the axes of the information set arrangement 
space and resetting the information set arrangement 
space with one or more axes, and the feature value ex- 
tracting portion extracts the component of the feature 
value based on the axis-resetting by the axis setting por- 
tion, the information set arranging portion arranges the 
information sets in the information set arrangement 
space based on the component of the extracted feature 
value, and the information displaying portion displays 
the pieces of media information corresponding to the 
viewpoint with respect to the reset information set ar- 
rangement space. 

[0022] With the above configuration, after seeing the 
result of the retrieval executed by a searcher, it is pos- 
sible to narrow down information flexibly by carrying out 
another retrieval in an interactive manner and carry out 
one retrieval after another by a further aspect using the 
reset information set arrangement space axes. In other 
words, the searcher can see the retrieval result, reset 
the axis of the information set arrangement space by 
specifying another feature value, rearrange and redis- 
play the information sets using the information set ar- 
rangement space whose axes have been reset, by trial 
and error. 

[0023] Next, in the multimedia information arranging 



apparatus described above, it is preferable that the axis 
setting portion resets an assignment of the attribute of 
the feature value to each of the axis of the information 
set arrangement space and resets the information set 

5 arrangement space that has been displayed already, the 
information set arranging portion rearranges the infor- 
mation sets in the reset information set arrangement 
space, and when displaying how the information sets 
are rearranged, the information displaying portion 

f 0 moves the displayed pieces of media information at pre- 
determined intervals from a position at which the infor- 
mation sets have been located before the rearrange- 
ment to a position at which they are to be located there- 
after. 

is [0024] With the above configuration, when resetting 
the information set arrangement space and reclassify- 
ing the information sets, it is possible to recognize vis- 
ually how the arrangement position of each information 
set changes, thus improving convenience of retrieval 

20 and classification operation of the information set. 
[0025] Furthermore, in the multimedia information ar- 
ranging apparatus described above, it is preferable that 
the information set arranging portion has a function of 
fixing an information set selected by a user to a specific 

25 position in an information set arrangement space spec- 
ified by the user, and a function of fixing the information 
set selected by the user to the specific position while 
rearranging only the other information sets according to 
the information set arrangement space, when rearrang- 

30 ing the information sets with respect to the information 
set arrangement space with reset axis, 
[0026] With the above configuration, since the display 
position of a target information set is fixed to a specific 
position, it is easy to find the target information set. Also, 

35 since the target information set and information sets 
similar to the target information set in the feature value 
that is set to the axis are arranged close to each other, 
it becomes easier to grasp the relationship between the 
information sets. Furthermore, it is possible to execute 

^0 a trial-and-error reclassification/redispiaying while fo- 
cusing on a specific information set. 
[0027] Next, a multimedia information arranging ap- 
paratus of the present invention can be realized using 
computers by recording a processing program of the 

^5 multimedia information arranging apparatus of the 
present invention, on a computer-readable recording 
medium. 

[0028] Then, a multimedia information arranging 
method of the present invention includes obtaining piec- 

50 es of media information in units of information sets, each 
formed by associating with each other related pieces of 
media information including image information, text in- 
formation and audio information, the media information 
being of a same kind and different kinds, assigning an 

55 attribute of a feature value extracted from the obtained 
pieces of media information contained in the information 
sets to an axis of a space in which a group of the infor- 
mation sets is arranged and setting an information set 
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arrangement'space with one or more axis, extracting a 
component of the feature value from the pieces of media 
information in the information sets, arranging the infor- 
mation sets in the information set arrangement space 
based on the attribute of the feature value of the pieces 
of media information contained in the information sets 
and the component of this feature value, specifying a 
feature value that is different from the feature value with 
respect to the arranged information sets, resetting an 
assignment to each of the axis of the information set ar- 
rangement space, and rearranging the information sets 
in the information set arrangement space based on the 
resetting, and setting the axis of the information set ar- 
rangement space and arranging the information sets in 
the information set arrangement space repeatedly while 
switching feature values to be used, thereby arranging 
the information sets. 

[0029] With the above configuration, the multimedia 
information can be retrieved by setting the axis of the 
information set arrangement space and arranging the 
information sets in the information set arrangement 
space repeatedly while switching feature values to be 
used. In the conventional retrieval method, when a re- 
trieval key that had been used was not sufficient for the 
retrieval narrowing-down, another retrieval was carried 
out by selecting indeterminately another retrieval key 
that belonged to the same kind and the same feature 
value. On the other hand, in the arranging method of the 
present invention, when a retrieval key that has been 
used is not sufficient for the retrieval narrowing-down, 
another retrieval can be carried out by selecting suitably 
a retrieval key that belongs to a different kind and a dif- 
ferent feature value, allowing a more flexible and appro- 
priate retrieval. In addition, since a retrieval key that be- 
longs to a different kind and a different feature value is 
used as described above, it can be expected that a heu- 
risticaily new retrieval result that a searcher has never 
expected can be obtained. For example, if a first retriev- 
al is executed with a feature value representing a pres- 
ence of a specific word with respect to the text informa- 
tion, and then these information sets are rearranged by 
combining a wavelet transform coefficient feature value 
and a HSI color histogram feature value with respect to 
the image information, it is possible to provide a new 
application of the retrieval system for discovering a ten- 
dency that has not been known conventionally, for ex- 
ample., information sets retrieved by a specific word tend 
to have specific partial shape and color. 

Brief Description of Drawings 

[0030] 

Fig. 1 is a drawing for describing a concept of an 
information set as a processing unit used in a mul- 
timedia information arranging apparatus of the 
present invention. 

Fig. 2 is a drawing illustrating an exemplary config- 



uration of a multimedia information arranging appa- 
ratus of a first embodiment according to the present 
invention. 

Fig. 3 is a flowchart showing an operation sequence 
of the multimedia information arranging apparatus 
of the first embodiment according to the present in- 
vention. 

Fig. 4 is a photograph showing a display example 
when information sets collected with a keyword of 
"bag" are arranged in an information set arrange- 
ment space, in the multimedia information arrang- 
ing apparatus of the first embodiment according to 
the present invention. 

Fig. 5 is a photograph showing an example in which 
ID numbers of text information attached to image 
information in the information set arrangement 
space shown in Fig. 4 are displayed by a pull-down 
menu. 

Fig. 6 is a photograph showing an example in which 
an ID number of the text information is selected from 
the pull-down menu shown in Fig. 5 and then a cor- 
responding text is displayed. 
Fig. 7 is a photograph showing an example in which 
information set arrangement space axes are reset 
by an axis setting portion 20, feature values are ex- 
tracted again, and then reclassification and redis- 
playing in this information set arrangement space 
are conducted. 

Fig. 8 is a photograph showing an example in which 
pieces of image information similar to a key image 
are displayed as a similar image list. 
Fig. 9 is a drawing illustrating an exemplary config- 
uration of a multimedia information arranging appa- 
ratus of a second embodiment according to the 
present invention. 

Fig. 10 is a drawing illustrating an exemplary con- 
figuration of a multimedia information arranging ap- 
paratus of a third embodiment according to the 
present invention. 

Fig. 11 is a drawing illustrating an exemplary con- 
figuration of a multimedia information arranging ap- 
paratus of a fourth embodiment according to the 
present invention. 

Fig. 12 is a photograph showing a display example 
when a display viewpoint is made to advance to- 
ward a depth direction, in a multimedia information 
arranging apparatus of a fifth embodiment accord- 
ing to the present invention. 
Fig. 13 is a drawing illustrating an exemplary con- 
figuration of the multimedia information arranging 
apparatus of the fifth embodiment according to the 
present invention. 

Fig. 14 is a view showing a specific example dis- 
played by an information displaying portion 50 be- 
fore rearrangement. 

Fig. 15 is a view showing the specific example dis- 
played by the information displaying portion 50 after 
the rearrangement. 
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Fig. 1 6 j« a view showing a state in which a certain 
information set is selected and specified using an 
arrangement position fixing specifying portion 42 
before rearrangement. 

Fig. 1 7 is a view showing a state in which the infor- 5 
mation set selected and specified in Fig. 1 6 is fixed 
to a specific position (at the center of a screen) while 
other information sets are rearranged. 
Fig. 1 8 is a drawing illustrating an example of con- 
structing a multimedia information arranging appa- 10 
ratus of a sixth embodiment according to the 
present invention into a client/ server configuration. 
Fig. 19 is a drawing illustrating an example of con- 
structing the multimedia information arranging ap- 
paratus of the sixth embodiment according to the 15 
present invention into a client/ server configuration. 
Fig. 20 is a drawing illustrating an example of con- 
structing the multimedia information arranging ap- 
paratus of the sixth embodiment according to the 
present invention into a client/ server configuration. 20 
Fig. 21 is a drawing illustrating an exemplary re- 
cording medium on which a program containing, as 
its processing operation, a processing content of a 
multimedia information arranging apparatus of a 
seventh embodiment according to the present in- 25 
vention is recorded. 

Best Mode for Carrying Out the Invention 

[0031] The following is a description of embodiments 30 
of a multimedia information arranging apparatus and an 
arranging method of the present invention, with refer- 
ence to the accompanying drawings. 

(First Embodiment) 35 

[0032] A multimedia information arranging apparatus 
of a first embodiment of the present invention will be de- 
scribed. Considering a group of related pieces of media 
information including image information, text informa- *o 
tion and audio information as an information set, the 
multimedia information arranging apparatus of the first 
embodiment arranges such information sets in a preset 
information set arrangement space and displays the 
pieces of media information according to a viewpoint 45 
with respect to this information set arrangement space. 
[0033] First, a concept of the information set, which 
serves as a unit of processing information used in the 
multimedia information arranging apparatus of the 
present invention, will be described. Next, an exemplary so 
configuration of the multimedia information arranging 
apparatus of the first embodiment will be described, and 
then an operation sequence of the multimedia informa- 
tion arranging apparatus of the first embodiment will be 
described with reference to a flowchart. 55 
[0034] The multimedia information arranging appara- 
tus of the present invention uses the information set as 
the unit of processing information. This information set 



is a processing unit obtained by associating with each 
other related pieces of media information such as image 
information, text information and audio information, the 
media information being of the same kind and different 
kinds. Fig. 1 is a schematic view showing this concept 
in a simplified manner. As shown in Fig. 1 , in one infor- 
mation set 1 , related pieces of media information, which 
is of the same kind or different kinds, are associated with 
each other. In the example of the information set 1 , four 
pieces of media information of the same kind or different 
kinds, which are an image information 1 a and an image 
information 1b, an audio information 1c and a keyword 
information 1d, are associated with each other. These 
four pieces of media information, which is of the same 
kind or different kinds, are collected based on a concept 
that these pieces of information are related to, for ex- 
ample, "a personal computer of Company F." 
[0035] When associating image information and text 
information with each other as an information set, for 
example, from a HTML document on the WWW in which 
images and texts are mixedly present, an image part is 
extracted as the image information, texts around the im- 
age in the HTML document are extracted as the text in- 
formation, and then they are associated with each other. 
As another example, when associating image informa- 
tion, audio information and text information in moving 
images with each other, from an XML file including mov- 
ie data containing moving images and audio, a moving 
image part is extracted as the moving image informa- 
tion, audio data are extracted as the audio information, 
texts around parts in which the movie data are embed- 
ded are extracted as the text information, and then they 
are associated with each other. Alternatively, it may be 
possible to trace a link provided in HTML data to other 
data and extract image information, text information and 
audio information from the linked content so as to be an 
information set. Also, there can be cases not only where 
image information and text information are originally 
present in the form of one file as in the HTML document, 
but also where plural pieces of text information are as- 
sociated with one piece of image information, or con- 
versely, plural pieces of image information are associ- 
ated with one text. In addition, URL (Universal Resource 
Locator) can be included as a part of the information set. 
By including URL as a part of the information set, it be- 
comes possible to select an arranged information set 
and display a Web page including this information set 
based on that URL as described below. 
[0036] One of the characteristics of this information 
set is that, while individual pieces of media information 
therein are not edited or processed so that image infor- 
mation is maintained as image information and audio 
information is maintained as audio information, each 
kind of feature values of the individual pieces of media 
information can be processed as a feature value of the 
entire information set. When arranging the information 
set based on a feature value as a processing unit as 
described below, each kind of feature values of the in- 



11 



EP 1 241 585 A1 



12 



dividual pieces of medra information in the information 
set is processed as the feature value of the entire infor- 
mation set, thereby determining a position at which the 
information set is to be arranged. On the other hand, 
when displaying the information set as described below, 
in the case where an image is displayed on an XY plane 
of a display screen (where a Z axis corresponds to a 
depth direction), one or more pieces of image informa- 
tion of the information set are displayed at the display 
position. If text information is displayed on the XY plane 
of the display screen, one or more pieces of text infor- 
mation of the information set are displayed at the display 
position of the individual information set. 
[0037] Using the information set of the present inven- 
tion as a processing unit as above is clearly different 
from a conventional multimedia processing. The con- 
ventional multimedia processing mainly refers to the fol- 
lowing two processings. The conventional multimedia 
processing first means that one apparatus can deal with 
a plurality of media, which are image information, text 
information and audio information. In this case, although 
one apparatus can deal with a plurality of media, the 
processing unit itself is directed to each piece of media 
information. That is, each piece of the media information 
such as the image information, the text information and 
the audio information is processed individually. The con- 
ventional multimedia processing secondly means that 
the processing unit itself is multimedia data into which 
a plurality of media have been integrated. In this case, 
the image information and the audio information are em- 
bedded or a link thereof is provided in the text informa- 
tion, for example. That is, the data themselves are inte- 
grally processed. On the other hand, the information set 
of the present invention formed by associating with each 
other related pieces of media information such as the 
image information, the text information and the audio in- 
formation, the media information being of the same kind 
and different kinds, is used as a processing unit and is 
different from the case where each piece of media infor- 
mation is processed individually as in the firstly-de- 
scribed conventional multimedia information. Further- 
more, although data of plural pieces of media informa- 
tion are associated so as to be a set in the present in- 
vention, the data of the plural pieces of media them- 
selves are not integrally processed unlike the secondly- 
described conventional multimedia information, but 
rather each piece of the media information is maintained 
as they are collected and obtained. Also, it becomes 
possible to incorporate newly added media information 
into an existing information set related to this new infor- 
mation in a simplified manner or to form a new informa- 
tion set when there is no existing information set that is 
related to the new information. If the data of the plural 
pieces of media themselves are integrally processed as 
in the secondly-described conventional multimedia in- 
formation, new media information cannot be added in- 
dependently and easily. 

[0038] Next, Fig. 2 shows an exemplary configuration 



of the multimedia information arranging apparatus of the 
first embodiment. As shown in Fig. 2, the multimedia in- 
formation arranging apparatus includes an information 
set obtaining portion 10, an axis setting portion 20, a 

5 feature value extracting portion 30, an information set 
arranging portion 40 and an information displaying por- 
tion 50. Numeral 60 denotes an accessible network 
such as the internet, and a multimedia information 
source 70 is being accessible via the network 60. In the 

io multimedia information source 70, various pieces of 
multimedia information such as image information, text 
information and audio information are stored. In this ex- 
ample, information sets, each of which is a group of re- 
lated pieces of media information, are also stored. 

15 [0039] In the exemplary configuration shown in Fig. 2, 
the information set obtaining portion 10 includes an in- 
formation set input portion 1 1 and an information set col- 
lecting portion 12. The information set collecting portion 
12 is a portion for collecting information sets, which col- 

20 lects information sets that are present in the multimedia 
information source 70 via the network 60. The informa- 
tion set input portion 1 1 can accept inputs of information 
sets directly from a recording medium such as a 
CD-ROM. As described above, the information set ob- 

25 taining portion 1 0 suitably includes either or both of the 
information set input portion 11 and the information set 
collecting portion 12, thereby holding selectively a func- 
tion of collecting information sets that are present in the 
multimedia information source 70 via the network 60 

30 and a function of accepting inputs of information sets 
directly from a recording medium such as a CD-ROM. 
[0040] An exemplary configuration of the information 
set collecting portion 12, which will be described in the 
first embodiment, includes a communication interface 

35 13, a recording medium 14 such as a hard disk and an 
information set collecting key input portion 15. Commu- 
nication is conducted with the multimedia information 
source 70 on the network 60 via the communication in- 
terface 13. The recording medium 14 can be used for 

40 storing the collected information sets. The information 
set collecting key input portion 1 5 specifies a collection 
range using a keyword at the time of collecting the in- 
formation sets. When a large amount of information sets 
is stored in the multimedia information source 70 on the 

^5 network, the amount of collected data becomes ex- 
tremely large if information sets are collected without 
specifying any range. Thus, when a keyword is inputted 
through the information set collecting key input portion 
15 so as to narrow down the range before collecting in- 

50 formation sets, the information set collecting portion 12 
collects the information sets having this keyword. 
[0041] The axis setting portion 20 is a portion for set- 
ting information set arrangement space axes, which as- 
signs a feature value extracted from each piece of the 

55 media information to each information set arrangement 
space axis and sets an information set arrangement 
space having one or more axes. For example, it speci- 
fies three axes of an X axis, a Y axis and a Z axis and 
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sets the space defined by these X, Y and Z axes as the 
information set arrangement space. In the present em- 
bodiment, a display screen of the information displaying 
portion 50 described below corresponds to the XY plane 
and a depth direction thereof corresponds to the Z-axis 
direction, for example. 

[0042] The feature value that is set as the information 
set arrangement space axis can be any feature value 
extractable according to media such as image informa- 
tion, text information and audio information. 
[0043] For example, with respect to the image infor- 
mation, the feature value may be a DCT coefficient fea- 
ture value, a wavelet transform coefficient feature value 
or a HSI color histogram feature value. By setting the 
DCT coefficient feature value as the information set ar- 
rangement space axis, it becomes possible to arrange 
the image information according to a feature of a spatial 
frequency component. By setting the wavelet transform 
coefficient feature value as the information set arrange- 
ment space axis, it becomes possible to arrange the im- 
age information according to a feature of a particularly 
low frequency portion of the spatial frequency, that is, a 
feature of a general outline of an object in the image. 
Although the wavelet transform also is a waveform/ fre- 
quency transform as the DCT, it can be performed while, 
maintaining positional (time) information. By setting the 
HSI color histogram feature value as the information set 
arrangement space axis, it becomes possible to arrange 
the information according to color information of the im- 
age. The HSI color histogram allows a better grasp of a 
feature of the image such as an extent to which a human 
skin region is included. 

[0044] Also, for example, with respect to the text in- 
formation, the feature value may be a feature value rep- 
resenting a presence of a specific word or a feature val- 
ue of how many times a specific word is used. By setting 
the feature value representing the presence of a specific 
word or the feature value of how many times a specific 
word is used as the information set arrangement space 
axis, it becomes possible to arrange the text information 
containing a description of the specific word. When im- 
age information is associated with the text information 
in the information set, the image information represent- 
ed by the specific word also is arranged in the informa- 
tion set arrangement space. 

[0045] In addition, for example, with respect to the au- 
dio information, the feature value may be a voice fre- 
quency feature value, an audio amplitude feature value 
or an audio time change feature value. By setting the 
voice frequency feature value as the information set ar- 
rangement space axis, it becomes possible to arrange 
the audio information according to a feature of the voice 
frequency, that is, audio pitch and quality. The voice fre- 
quency makes it possible to indicate the feature of the 
audio information such as a difference in sounding ob- 
jects and a difference between an animal bark, a male 
voice and a female voice, and with an improved accu- 
racy, to indicate a difference in speakers. By setting the 



audio amplitude feature value or the time change fea- 
ture value as the information set arrangement space ax- 
is, it becomes possible to arrange the audio information 
according to an audio volume. 

5 [0046] The axis setting portion 20 also can assign a 
combination of a plurality of feature values to one space 
axis. When combining two or more feature values, units 
of these feature values have to be transformed and ad- 
justed, and in such cases, they can be converted into 

10 points such as scores and then combined. For example, 
the case in which a specified color component of the 
HSI color histogram is contained at a ratio equal to or 
more than a threshold is expressed by "1," and other- 
wise by "0" as a first feature value, and the case in which 

'5 there is a feature value representing the presence of a 
specific word in the text information is expressed by "1 ," 
and otherwise by "0" as a second feature value, where- 
by the first feature value and the second feature value 
can be combined and assigned to one space axis. Con- 

20 versely, one feature value also can be assigned to a plu- 
rality of the axes. There are many methods for assigning 
one feature value to two or more axes, and one of them 
is to regard the feature value as a vector, select a plu- 
rality of dimensions of the vector that have a large var- 

25 iance and define these dimensions as the axes. 

[0047] The feature value extracting portion 30 ex- 
tracts a feature value from each of the media information 
of the information set. Although not shown in Fig. 2, the 
feature value extracting portion 30 has a function of ex- 

30 tracting various feature values from the media informa- 
tion as described above. For example, it has a DCT co- 
efficient feature value calculating function, a wavelet 
transform coefficient feature value calculating function 
and a HSI color histogram feature value calculating 

35 function as a function of extracting a feature value of the 
image information. For example, it has a function of de- 
tecting whether or not a specific word is present and a 
function of calculating how many times a specific word 
is used as a function of extracting a feature value of the 

40 text information. For example, it has a voice frequency 
feature value extracting function, an audio amplitude 
feature value extracting function and ah audio time 
change feature value extracting function as a function 
of extracting a feature value of the audio information. By 

^5 utilizing these functions, a feature value is extracted 
from each piece of the media information of the collected 
information set. Furthermore, the addition, update and 
deletion of the function of extracting the feature value 
from each piece of the media information are possible 

so in the feature value extracting portion 30. When the fea- 
ture value extracting function is provided as DSP (digital 
signal processor) or the like, thecontent can be replaced 
and added easily as necessary. 
[0048] The information set arranging portion 40 ar- 

55 ranges the information sets in the information set ar- 
rangement space based on the feature values extracted 
by the feature value extracting portion 30. For example, 
when the information set arrangement space is set by 
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the three axes of X, Y*and Z, the information sets are 
arranged in this three-dimensional information set ar- 
rangement space. 

[0049] The information displaying portion 50 is a por- 
tion for displaying pieces of the media information in the 
information sets that have been arranged in the infor- 
mation set arrangement space by the information set ar- 
ranging portion 40. The information displaying portion 
50 displays pieces of the media information in the infor- 
mation set arrangement space from a direction accord- 
ing to a viewpoint with respect to the information set ar- 
rangement space. For example, when the XY plane is 
a front surface and the Z-axis direction is a depth direc- 
tion in the information set arrangement space defined 
by the X, Y and Z axes, the media information is dis- 
played such that the display screen corresponds to the 
XY plane and the depth direction of the screen corre- 
sponds to the Z-axis direction. 
[0050] An example of the operation sequence of the 
multimedia information arranging apparatus of the first 
embodiment of the present invention configured as 
above will be described with reference to Fig. 3. 
[0051] As shown in Fig. 3, the operation of the multi- 
media information arranging apparatus of the first em- 
bodiment of the present invention generally includes an 
information set obtaining operation (S101), an axis set- 
ting operation (S102) for setting a feature value to be 
assigned to an information set arrangement space axis 
and defining an information set arrangement space, a 
feature value extracting operation (S103) for extracting 
predetermined feature values from pieces of media in- 
formation in the information sets, an information arrang- 
ing operation (S104) for arranging the information sets 
in the information set arrangement space according to 
the extracted feature values, an information displaying 
operation (S105) for displaying the information set ar- 
rangement space and the information sets that have 
been arranged in the information set arrangement 
space from a set viewpoint, and an axis resetting oper- 
ation (a loop to Operation S1 02) for seeing the displayed 
retrieval result, resetting the feature value to be as- 
signed to the information set arrangement space axis 
and redefining the information set arrangement space, 
in order to narrow down further or continue retrieval from 
another aspect as necessary. When a desired retrieval 
result is obtained by arranging the information sets 
based on the information set arrangement space that 
has been set by the first axis setting operation (S102), 
the axis resetting operation does not have to be con- 
ducted. Although the above operation sequence pre- 
supposes that the information sets are present acces- 
sibly from the beginning, the multimedia information ar- 
ranging apparatus may execute, as a preprocessing, an 
information set generating operation for grouping to- 
gether related pieces of the media information including 
the image information, the text information and the audio 
information and defining/ generating the information 
sets, as described in the second embodiment. 



[0052] First, the multimedia information arranging ap- 
paratus of the present invention executes the informa- 
tion set obtaining operation (S101) by the information 
set obtaining portion 10. For example, the information 

5 set obtaining portion 10 collects information sets from 
the multimedia information source 70 on a WWW server 
or the like, which is accessible on the network 60 such 
as the internet, via the communication interface 13, thus 
obtaining the information sets. In this example, as the 

10 information set obtaining operation (S1 01 ), a plurality of 
the information sets in which image information and text 
information are associated with each other are obtained. 
Also, in order to narrow down the range of information 
sets to be obtained to a certain degree, only the infor- 
ms mation sets that are hit in a keyword retrieval may be 
obtained. In this example, the information sets that are 
hit by the keyword of "bag" were obtained. 
[0053] Next, the multimedia information arranging ap- 
paratus executes the axis setting operation (S102) for 

20 setting a feature value to be assigned to the information 
set arrangement space axis and defining an information 
set arrangement space by the axis setting portion 20. 
The axis setting portion 20 executes the axis setting op- 
eration by setting the feature value to be assigned to 

25 each axis of the information set arrangement space 
among feature values extractable from the media infor- 
mation such as the DCT coefficient feature value with 
respect to the image information described above and 
defining the information set arrangement space. In this 

30 example, the wavelet transform coefficient feature value 
of the image information is assigned to an X axis (a hor- 
izontal direction), the HSI color histogram feature value 
is assigned to a Y axis (a vertical direction), and the fea- 
ture value representing how many times a specific word 

35 is used in the text information is assigned to a Z axis (a 
depth direction). In the present description, the feature 
value representing the presence of a specific word, 
which is to be assigned to the Z axis, is the number of 
times that the word "bag" is used when collecting the 

40 information sets. 

[0054] Then, the multimedia information arranging 
apparatus executes the feature value extracting opera- 
tion (S103) for extracting the feature values that have 
been assigned to the space axes from the pieces of the 

45 media information in the collected information sets using 
the feature value extracting portion 30. As described 
above, although not shown in Fig. 2, the feature value 
extracting portion 30 has the wavelet transform coeffi- 
cient feature value calculating function, the HSI color 

so histogram feature value calculating function and the 
function of calculating how many times a specific word 
is used, and extracts the wavelet transform coefficient 
feature value, the HSI color histogram and the number 
of times that the specific word is used from, the media 

55 information of the collected information sets. 

[0055] Subsequently, the multimedia information ar- 
ranging apparatus executes the information arranging 
operation (S104) for arranging the information sets in 
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the information set arrangement space according to the 
extracted feature values using the information set ar- 
ranging portion 40. Then , it executes the information dis- 
playing operation (S105) for displaying the information 
set arrangement space and the information sets that 5 
have been arranged in the information set arrangement 
space from the set viewpoint by the information display- 
ing portion 50. Fig. 4 shows an example of arranging the 
information sets collected by the keyword of "bag" in the 
set information set arrangement space. In this example, 
each information set is displayed such that the image 
information among the associated pieces of the media 
information is located at the front (the XY plane). Since 
the X axis is the wavelet transform coefficient feature 
value, bags with similar shapes are located in similar 
positions on the X coordinate. Since the Y axis is the 
HSI color histogram feature value, bags with similar 
colors are located in similar positions on the Y coordi- 
nate. 

[0056] In some information sets that have been ar- 
ranged in the information set arrangement space as 
shown in Fig. 4, plural pieces of the text information are 
associated. The image information displayed at the front 
in Fig. 4 is accompanied by text information. In this case, 
as shown in Fig. 5, when the image information on the 
information set arrangement space is clicked with a 
pointing device, ID numbers (for example, "text 1 H ) of 
the accompanying text information are displayed by a 
pull-down menu. When an ID number of the text infor- 
mation is selected from the pull-down menu, a corre- 
sponding text is displayed as shown in Fig. 6. 
[0057] According to the information set obtaining op- 
eration S101 to the information displaying operation 
S105 described above, the multimedia information ar- 
ranging apparatus of the present invention can end the 
retrieval operation when desired image information is 
obtained, the initial object of the retrieval operation is 
achieved, and thus an axis resetting operation is not 
necessary (Operation S106: NO), while it can return 
along the loop to Operation S 102 to perform the axis 
resetting operation when the initial object of the retrieval 
operation has not been achieved, and the axis resetting 
operation is necessary (Operation S106: YES). In other 
words, the multimedia information arranging apparatus 
executes the axis resetting operation of seeing the ar- 
ranged result displayed in the information displaying op- 
eration S105, resetting the feature value to be assigned 
to the information set arrangement space axis by using 
the axis setting portion 20 and redefining the information 
set arrangement space, in order to narrow down further 
or continue retrieval from another aspect as necessary. 
As described above, until the needed image information 
is obtained, it resets the feature values, to be assigned 
to the information set arrangement space axes (S102), 
extracts the reset feature values again from the pieces 
of the media information in the information sets (S103), 
rearrange the information sets in the information set ar- 
rangement space based on the newly extracted feature 



values (S104) and redisplays the information sets that 
have been rearranged in the information set arrange- 
ment space by the information displaying portion 50 
(S1 06). For example, the axis setting portion 20 also can 
change the X axis from the wavelet transform coefficient 
feature value to the DOT coefficient feature value or 
change the Y axis from the HSI color histogram feature 
value to the voice frequency feature value as a totally 
different aspect. In this example, the X axis was 
changed from the wavelet transform coefficient feature 
value to the HSI color histogram feature value, and the 
Y axis was changed from the HSI color histogram fea- 
ture value to the DCT coefficient feature value. Fig. 7 
shows an example in which the axis setting portion 20 
resets the information set arrangement space axes and 
extracts the feature values again, followed by reclassi- 
fication and redisplaying in this information set arrange- 
ment space. A series of these axis resetting operations 
are repeated until a desired retrieval result is obtained. 
[0058] As described above, it is possible to see the 
displayed arranged result of the information sets ob- 
tained with certain feature values and try another ar- 
rangement of the information sets using feature values 
with a completely different aspect, thus generating a 
possibility of discovering new information that a search- 
er has never expected. 

[0059] It also is possible to see the displayed results 
as the arrangement of the information sets such as the 
image information and process the information as de- 
scribed below. 

[0060] First, a Web page containing an information 
set selected from the arranged information sets can be 
displayed. If URL including the information set such as 
the image information also is stored in a part of the in- 
formation set, in the case where this image information 
on the information displaying portion 50 is selected by 
a user, its Web page can be displayed based on the URL 
information. For example, image information is select- 
ed, and then a button of "Web page" in the menu is 
clicked with a pointing device, thereby displaying this 
Web page. 

[0061] Second, by seeing the arranged information 
sets so as to provide a retrieval key, information sets 
similar to this key can be displayed as a list. For exam- 
ple, with respect to the information sets arranged as in 
Fig. 4, a key image (an image to be a key for pattern 
matching) is provided as the retrieval key, and pieces of 
image information, which is considered similar to this 
key image by the pattern matching, are displayed as a 
similar image list. Fig. 8 shows this example. The similar 
image list is displayed with respect to the input key im- 
age. The example of Fig. 8 provides the key image as 
the retrieval key, but it also may be possible to provide 
text information as the retrieval key. In this case, an input 
keyword thereof and the similar image list are displayed. 
[0062] As described above, according to the multime- 
dia information arranging apparatus of the first embod- 
iment, considering a group of related pieces of the me- 
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dia informatiQn including the image information, the text 
information and the audio information as an information 
set, it is possible to arrange such information sets in a 
preset information set arrangement space and display 
the media information according to a viewpoint with re- 
spect to this information set arrangement space. 

(Second Embodiment) 

[0063] A multimedia information arranging apparatus 
of a second embodiment of the present invention will be 
described. The multimedia information arranging appa- 
ratus of the second embodiment has a function of col- 
lecting pieces of media information including the image 
information, the text information and the audio informa- 
tion, analyzing the relationship between the collected 
pieces of the media information, grouping together re- 
lated pieces of the media information so as to generate 
information sets, as a preprocessing before the informa- 
tion set arranging operation described in the first em- 
bodiment. 

[0064] First, Fig. 9 shows an exemplary configuration 
of the multimedia information arranging apparatus of the 
second embodiment. As shown in Fig. 9, in the multi- 
media information arranging apparatus of the second 
embodiment, elements other than the information set 
obtaining portion 1 0, namely, the axis setting portion 20, 
the feature value extracting portion 30, the information 
set arranging portion 40, the information displaying por- 
tion 50, the network 60 and the multimedia information 
source 70 may be the same as those in the exemplary 
configuration of the multimedia information arranging 
apparatus shown in Fig. 2 described in the first embod- 
iment. 

[0065] As shown in Fig. 9, in the multimedia informa- 
tion arranging apparatus of the second embodiment, the 
information set collecting portion 12 of the information 
set obtaining portion 10 includes an information collect- 
ing portion 14, a relationship analyzing portion 15 and 
an information set generating portion 16 in addition to 
the communication interface 13, the recording medium 
14 and the information set collecting key input portion 
15. 

[0066] The information collecting portion 14 collects 
media information including image information, text in- 
formation and audio information stored in the accessible 
multimedia information source 70 on the network 60. 
The media information can be collected automatically 
using a robot. When using a robot, a selection criterion 
is specified for collecting the media information from the 
multimedia information source 70 on the network 60. For 
example, the criterion is selected from a criterion group 
including keyword information, site information, link in- 
formation and similarity information with respect to a 
specific information set. When providing the keyword in- 
formation as the selection criterion, the media informa- 
tion without this keyword is not collected, so that the 
range can be limited. A text retrieval server is supplied 



with a keyword, so that the resultant feedback pages 
are retrieved. 

[0067] When providing the site information and the 
link information as the selection criteria, the robot re- 

s trieves pages corresponding to a specified URL and 
thereafter and pages to which the URL is linked. In this 
manner, by circulating on the WWW and tracing the 
links, the robot traces a plurality of Web pages. 
[0068] When providing the similarity information with 

10 respect to a specific information set as the selection cri- 
terion, it is possible to collect mainly the media informa- 
tion similar to media information in a certain the infor- 
mation set. 

[0069] The information collecting portion 14 collects 
15 the image information, the text information and the audio 
information and stores them in the recording medium 
14. 

[0070] The relationship analyzing portion 1 5 analyzes 
the relationship between pieces of the media informa- 

20 tion collected from the multimedia information source by 
the information collecting portion 1 4. For example, when 
analyzing a text related to image information, in the case 
of a HTML document, the relationship analyzing portion 
15 interprets a HTML structure while referring to texts 

25 around this image and information of the HTML, extracts 
an image part as the image information, extracts the 
texts around this image in the HTML document as the 
text information, and then analyzes a related degree of 
the texts near the image, it also is possible to analyze 

30 the related degree of the audio information in a similar 
manner. Also, in the case of a file other than the HTML, 
the relationship can be analyzed considering that the im- 
age information, the text information and the audio in- 
formation in this file are highly related to each other, as 

35 long as they are integrated into one file such as a PDF 
file. In the case of where plural pieces of the media in- 
formation are not integrated into one file, as long as they 
are provided with the same keyword, it also is possible 
to analyze the relationship considering that the plural 

40 pieces of the media information are highly related to 
each other. If this keyword is not ordinary but distinctive, 
it also is possible to analyze the relationship considering 
that the related degree is still higher. Furthermore, it is 
needless to say that the related degree of plural pieces 

45 of the media information also can be determined by a 
user. As another example, in the case of a PDF file con- 
taining movie data including moving images and audio, 
a moving image part is extracted as the moving image 
information, audio data are extracted as the audio infor- 

50 mation, and texts around which the movie data are em- 
bedded are extracted as the text information, whereby 
these are associated with each other and formed into 
an information set. 

[0071] The information set generating portion 16 edits 
55 a grouping of related pieces of the media information as 
an information set, the media information being of the 
same kind and different kinds, based on the analysis re- 
sult by the relationship analyzing portion 15. 
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[0072] As described. above, the multimedia informa- 
tion arranging apparatus of the second embodiment can 
collect pieces of the media information including the im- 
age information, the text information and the audio in- 
formation, analyze the relationship between the collect- 5 
ed pieces of the media information, group together re- 
lated pieces of the media information so as to generate 
information sets, as a preprocessing before the informa- 
tion set arranging operation described in the first em- 
bodiment. Since the operation of arranging information 
sets in the information set arrangement space using the 
generated information sets is the same as that de- 
scribed with reference to the flowchart of Fig. 3 in the 
first embodiment, the detailed description thereof is 
omitted here. 

(Third Embodiment) 

[0073] A multimedia information arranging apparatus 
of a third embodiment of the present invention will be 
described. The multimedia information arranging appa- 
ratus of the third embodiment applies a self-organizing 
map in the arrangement in the information set arrange- 
ment space based on the feature value of the media in- 
formation by the information set arranging portion. 
[0074] Fig. 10 shows an exemplary configuration of 
the multimedia information arranging apparatus of the 
third embodiment. As shown in Fig. 10, the information 
set arranging portion 40 includes a self -organizing map 
processing portion 41 . Elements otherthan the informa- 
tion set arranging portion 40, namely, the information set 
obtaining portion 1 0, the axis setting portion 20, the fea- 
ture value extracting portion 30, the information display- 
ing portion 50, the network 60 and the multimedia infor- 
mation source 70 may be the same as those in the ex- 
emplary configuration of the multimedia information ar- 
ranging apparatus shown in Fig. 2 described in the first 
embodiment. 

[0075] The self-organizing map is a learning model 
without a teacher using a neural network. In the self- 
organizing map, a high-dimensional feature vector 
space is mapped to a low-dimensional space. At this 
time, data having similar feature vectors are arranged 
dose to each other also in the low-dimensional space. 
This self-organizing map is applied to the arranging op- 
eration of the media information, so as to arrange infor- 
mation sets using the self-organizing map processing 
based on feature values extracted by the feature value 
extracting portion 30. The self-organizing map process- 
ing portion 41 executes the self-organizing map gener- 
ating processing with respect to the feature values ex- 
tracted by the feature value extracting portion 80. The 
information set arranging portion 40 of the present em- 
bodiment performs an arrangement in the information 
set arrangement space based on arrangement informa- 
tion obtained by the self-organizing map generated by 
the self-organizing map processing portion 41 . It also is 
possible to combine the self-organizing map processing 



and a depth representation based on the feature value 
assigned to the Z axis (the depth direction). For exam- 
ple, the text information is decomposed into pieces of 
word frequency information, each frequency is vector- 
ized, and then an axis position in the depth direction is 
determined based on this vector. As another example, 
when the Web is searched using a keyword, based on 
a related degree between the keyword and a Web page 
fed back from the text retrieval server, the information 
sets can be displayed in descending order of the related 
degree toward the depth and further they can be dis- 
played by switching these methods. 
[0076] As described above, in accordance with the 
multimedia information arranging apparatus of the third 
embodiment, by applying the self-organizing map 
processing, images that are considered to have similar 
contents can be arranged closer to each other and im- 
ages that are considered to have dissimilar contents can 
be arranged far from each other in the space. 

(Fourth Embodiment) 

[0077] A multimedia information arranging apparatus 
of a fourth embodiment of the present invention will be 
described. The multimedia information arranging appa- 
ratus of the fourth embodiment is directed to contrivanc- 
es for a displaying method and a browsing method of 
the information set arrangement space in which the in- 
formation sets are arranged. 

[0078] Fig. 20 shows an exemplary configuration of 
the multimedia information arranging apparatus of the 
fourth embodiment. As shown in Fig. 1 1 , the information 
displaying portion 50 includes a display viewpoint mov- 
ing portion 51 . Elements otherthan the information dis- 
playing portion 50 ; namely, the information set obtaining 
portion 1 0, the axis setting portion 20, the feature value 
extracting portion 30, the information set arranging por- 
tion 40, the network 50 and the multimedia information 
source 70 may be the same as those in the exemplary 
configuration of the multimedia information arranging 
apparatus shown in Fig. 2 described in the first embod- 
iment. 

[0079] In the information displaying portion 50, the 
display viewpoint moving portion 51 has a function of 
moving a position of setting a viewpoint for displaying 
the information set arrangement space in which the in- 
formation sets are arranged by the information set ar- 
ranging portion 40. The information displaying portion 
50 displays the information set arrangement space seen 
from the display viewpoint set by the display viewpoint 
moving portion 51. 

[0080] In the first embodiment, an example of the in- 
formation set arrangement space displayed by the in- 
formation displaying portion 50 has been illustrated by 
Fig. 4. The multimedia information arranging apparatus 
of the third embodiment makes it possible to regard the 
display viewpoint of Fig. 4 as default and change the 
display viewpoint dynamically by the display viewpoint 
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moving portiQn 51 . In other words, the display viewpoint 
virtually can move freely in the information set arrange- 
ment space in which the information sets are arranged, 
making it possible to display how they are arranged in 
the information set arrangement space seen from its 
moved position. In general, since a display screen ba- 
sically is a two-dimensional plane, information sets lo- 
cated far in the depth tend to be difficult to see even 
though they can be displayed in perspective. However, 
the multimedia information arranging apparatus of the 
third embodiment can change the display viewpoint dy- 
namically, thus making it possible to display the arrange- 
ment state of the information sets that a searcher wants 
to see more closely, so as to be close to the display 
screen surface according to the searcher's operation. 
Fig. 1 2 shows a display example when the display view- 
point is made to advance from the state shown in Fig. 4 
toward a depth direction. 

(Fifth Embodiment) 

[0081] A multimedia information arranging apparatus 
of the fifth embodiment resets an information set ar- 
rangement space that has been displayed already by 
resetting an assignment of an attribute of a feature value 
to each information set arrangement space axis and re- 
arranges each information set with respect to the reset 
information set arrangement space. Then, when dis- 
playing how the information sets are rearranged, the 
multimedia information arranging apparatus of the 
present embodiment moves a displayed piece of media 
information at predetermined intervals from the position 
at which the information set is located before the rear- 
rangement to the position at which it is to be located 
thereafter. 

[0082] The multimedia information arranging appara- 
tus of the fifth embodiment also has a function of fixing 
an information set selected by a user to a specific posi- 
tion in the information set arrangement space specified 
by the user during the rearrangement and a function of 
fixing the information set selected by the user to the spe- 
cific position while rearranging only the other informa- 
tion sets according to the information set arrangement 
space when rearranging each information set with re- 
spect to the information set arrangement space whose 
axis has been reset. 

[0083] Fig. 1 3 shows a simplified exemplary configu- 
ration of the multimedia information arranging appara- 
tus according to the fifth embodiment. As shown in Fig. 
13, the information displaying portion 50 includes a mov- 
ing image processing portion 52. Furthermore, the in- 
formation set arranging portion 40 includes an arrange- 
ment position fixing specifying portion 42. Elements oth- 
er than the information set arranging portion 40 and the 
information displaying portion 50, namely, the informa- 
tion set obtaining portion 1 0, the axis setting portion 20, 
the feature value extracting portion 30, the network 60 
and the multimedia information source 70 may be the 



same as those in the exemplary configuration of the mul- 
timedia information arranging apparatus shown in Fig. 
2 described in the first embodiment. 
[0084] When displaying how the information sets are 
s rearranged in the information set arrangement space by 
resetting the axes, the moving image processing portion 
52 of the information displaying portion 50 has afunction 
of moving displayed media information in each informa- 
tion set at predetermined intervals from the position at 
10 which the media information is located before the rear- 
rangement to the position at which it is to be located 
thereafter. For example, the function includes storing 
the positions of the information sets before the rear- 
rangement, receiving a notification of the positions of 
'5 the information sets after the rearrangement from the 
information set arranging portion 40, calculating a mov- 
ing direction and a moving distance for each of the in- 
formation sets based on both the positions at the coor- 
dinates and moving displayed pieces of the media infor- 
mation in each information set at predetermined inter- 
vals. The predetermined interval may be a predeter- 
mined moving distance regardless of the number of 
moving steps or an interval obtained by adjusting a mov- 
ing distance so that the moving is completed within the 
predetermined number of steps. 
[0085] The information set arranging portion 40 in- 
cludes the arrangement position fixing specifying por- 
tion 42, and the user can specify via the arrangement 
position fixing specifying portion 42 that a specific infor- 
mation set is fixed to a specific position. When rearrang- 
ing each information set with respect to the information 
set arrangement space whose axis has been reset, the 
information set arranging portion 40 has a function of 
fixing a selected information set to a specific position 
while rearranging only the other information sets ac- 
cording to the information set arrangement space. 
[0086] The above-described rearranging functions of 
moving the displayed media information in the informa- 
tion set at predetermined intervals and fixing a specific 
information set at a specific position will be described 
with reference to specific examples of Figs. 14 and 15. 
[0087] First, Fig. 1 4 showing how the information sets 
are displayed in the information set arrangement space 
before the rearrangement will be described. In the infor- 
mation set arrangement space defined by X, Y and Z 
axes, the display screen of the information displaying 
portion 50 corresponds to the XY plane, while the depth 
direction of the screen corresponds to the Z axis. In the 
present embodiment, a HSI color histogram feature val- 
ue is assigned to the axis in the XY plane direction by 
the axis setting portion 20, while no particular feature 
value is assigned to the Z axis and "0" value is assigned 
to all. In this initial information set arrangement space 
before the rearrangement, the information set arranging 
portion 40 arranges each information set on the plane 
of Z = 0 according to the HSI color histogram of image 
media information contained in the information set. Fig. 
14 shows an example displayed by the information dis- 
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playing portiQn 50 before the rearrangement. In this ex- 
ample, each of the information sets contains image me- 
dia information of a pattern of fabric such as neckties. 
Accordingly, as shown in Fig. 1 4, images of the informa- 
tion sets that have similar color information of the image 
media information are arranged close to each other in 
the XY plane. 

[0088] The following is a description of how the rear- 
rangement is carried out. Each information set contains 
various kinds of keyword information such as a keyword 
representing its content and has a similarity to a specific 
keyword as a feature value. In the present embodiment, 
the feature value of the similarity to a specific keyword 
is assigned to the 2 axis by the axis setting portion 20. 
Subsequently, the information set arranging portion 40 
arranges the information sets according to the informa- 
tion set arrangement space whose axis is reset by the 
axis setting portion 20. In this case, the information sets 
that have been arranged in the XY plane before the re- 
arrangement are rearranged so that the feature value of 
the similarity to a specific keyword corresponds to the Z 
axis (the depth direction). Fig. 15 shows a specific ex- 
ample displayed by the information displaying portion 
50 after the rearrangement. In the example of Fig. 15, 
a "striped pattern" was used for the specific keyword. 
The result of the rearrangement of Fig. 15 was that the 
information sets were arranged in the XY plane direction 
based on the color histogram and in the Z direction 
based on the feature value of the specific keyword of 
the "striped pattern," so that the images of the informa- 
tion sets having the striped pattern were displayed en- 
larged at the front. In this manner, it becomes possible 
to visually grasp both a color tendency and a classifica- 
tion of the presence or absence of the striped pattern 
with respect to the group of information sets at the same 
time. Moreover, all the information sets were arranged 
in the XY plane before the rearrangement, whereas the 
information sets with a higher similarity to a specific key- 
word are displayed closer to the front and the ones with 
a lower similarity are displayed further to the back after 
the rearrangement. Accordingly, the information sets re- 
lated to a specific keyword can be made easier to see, 
whereas the ones not related very much thereto can be 
displayed small. 

[0089] In the present embodiment, the moving image 
processing portion 52 moves displayed media informa- 
tion in each information set at predetermined intervals 
from the position at which the media information is lo- 
cated before the rearrangement to the position at which 
it is to be located thereafter. In other words, it displays 
media information displayed in each information set at 
predetermined intervals from the position before the re- 
arrangement shown in Fig. 14 to the position thereafter 
shown in Fig. 15 as if the media information moves 
traceably by a human eye. The moving image process- 
ing portion 52 stores the positions of the information sets 
before the rearrangement shown in Fig. 1 4, receives the 
notification of the positions of the information sets after 



the rearrangement shown in Fig. 15, calculates the mov- 
ing direction and the moving distance for each informa- 
tion set based on both the positions at the coordinates 
and moves the displayed media information in each in- 
5 formation set at predetermined intervals. 

[0090] Furthermore, as another specific example, 
when the axis setting portion 20 assigns a wavelet trans- 
form coefficient feature value instead of the color histo- 
gram feature value to the X, Y axes, the information set 
arranging portion 40 arranges the information sets in the 
new information set arrangement space, and the infor- 
mation displaying portion 50 displays pieces of the me- 
dia information at predetermined intervals from the po- 
sition before the rearrangement to the position thereaf- 
ter as if they move traceably by human eyes, as de- 
scribed above. 

[0091] Next, the following description is directed to a 
specific example of an rearrangement while a specific 
information set is fixed to a specific position, using the 
arrangement position fixing specifying portion 42 of the 
information set arranging portion 40. 
[0092] In order to grasp the relationship between a 
specific information set and the other information sets, 
a user selects one information set or a plurality of infor- 
mation sets from the screen displaying these informa- 
tion sets via the arrangement position fixing specifying 
portion 42. For example, the user selects one informa- 
tion set and fixes it to the center, and then arranges the 
other information sets in the information set arrange- 
ment space by a self-organizing map method. It also is 
possible to select a plurality of information sets, for ex- 
ample, four information sets, and fix them to specific po- 
sitions, for example, four comers of the screen and then 
to arrange the other information sets in the information 
set arrangement space by the self-organizing map 
method. By carrying out the arrangement while fixing the 
specific information sets to the specific positions as de- 
scribed above, it becomes easier to grasp visually the 
relationship between the selected information sets and 
the other information sets. For example, in the case 
where an axis of the information set arrangement space 
has a feature value of color information, when carrying 
out the arrangement while fixing an red image, a blue 
image, a yellow image and a green image to the four 
corners respectively, reddish images gather at the cor- 
ner to which the red image is fixed and magenta images 
gather around the middle position between the position 
of the red image and that of the blue image. Therefore, 
it becomes easier to find individual images according to 
their hues. 

[0093] Figs. 16 and 17 show a concept of the rear- 
rangement while a specific information set is fixed to a 
specific position, using the arrangement position fixing 
specifying portion 42. Fig. 16 shows how the specific 
information set is selected and specified using the ar- 
rangement position fixing specifying portion 42 before 
the rearrangement, and Fig. 17 shows how the informa- 
tion set selected and specified in Fig. 16 is fixed to a 
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specific position (the center of the screen) and the other 
information sets are rearranged. In Fig. 16, an informa- 
tion set 5 corresponds to the information set selected 
and specified by a user via the arrangement position fix- 
ing specifying portion 42. When the rearrangement is 5 
carried out according to color information while this in- 
formation set 5 is fixed to the center, the information sets 
having a hue similar to that of the information set 5 gath- 
er around the center as shown in Fig. 1 7. 
[0094] The above description is merely an example, 
and other than the case of the color information, infor- 
mation sets having a text similar to that of the selected 
information set can be arranged in the vicinity of the se- 
lected information set when, for instance, a feature val- 
ue representing the presence of a specific word with re- 
spect to text information is assigned to the information 
set arrangement space axis. 

[0095] This rearrangement function while fixing the 
specific information set to the specific position using the 
arrangement position fixing specifying portion 42 can be 
combined with that of moving the displayed media infor- 
mation in each information set at predetermined inter- 
vals. Also, during the rearrangement while fixing the 
specific information set to the specific position, each in- 
formation set can be displayed in such a manner as to 
move to the position after the rearrangement at prede- 
termined intervals. 

[0096] With the above configuration, it is possible to 
reset the information set arrangement space and carry 
out a reclassification with a target information set being 
fixed to a specific position, allowing a visual recognition 
of how the arrangement position of each information set 
changes, thus improving the convenience of retrieval 
and classification operation of the information set. 

(Sixth Embodiment) 

[0097] A multimedia information arranging apparatus 
of the sixth embodiment of the present invention will be 
described. The multimedia information arranging appa- 
ratus of the sixth embodiment is obtained by construct- 
ing the above-described multimedia information arrang- 
ing apparatus of the first to fifth embodiments by a client/ 
server system via a computer network. Several patterns 
are possible depending on which elements are incorpo- 
rated into a server computer and which elements are 
incorporated into a client computer among the elements 
in the configuration of the above-described multimedia 
information arranging apparatus of the first to fifth em- 
bodiments. 

[0098] Fig. 1 8 illustrates an example of a client/ server 
configuration in which a server computer 1 00 is provided 
with the information set obtaining portion 1 0 and a client 
computer 101 is provided with the axis setting portion 
20, the feature value extracting portion 30, the informa- 
tion set arranging portion 40 and the information display- 
ing portion 50. 

[0099] Fig. 19 illustrates a configuration in which the 



server computer 1 00 is provided with the information set 
obtaining portion 10 and the feature value extracting 
portion 30 and the client computer 101 is provided with 
the axis setting portion 20, the information set arranging 
portion 40 and the information displaying portion 50. 
[0100] Fig. 20 illustrates a configuration in which the 
server computer 1 00 is provided with the information set 
obtaining portion 10, the feature value extracting portion 
30 and the information set arranging portion 40 and the 
client computer 101 is provided with the axis setting por- 
tion 20 and the information displaying portion 50. 
[0101] In each of the configurations of Figs. 1 8 to 20 
described above, there are several methods for obtain- 
ing information sets. For example, there is a method in 
which the server computer 1 00 serves as a robot to col- 
lect/ update automatically information sets having a 
specified content from the network in a periodic manner. 
Figs. 18 to 20 all illustrate the information set obtaining 
portion 1 0 incorporated in the server computer 1 00, but 
there also is another method in which the information 
set collecting key input portion 15 is separated from the 
information set collecting portion 12 of the information 
set obtaining portion 10 and then provided in the client 
computer 101 (not shown in the figures), a user of the 
client computer 101 inputs an information set collecting 
key using the information set collecting key input portion 
15, the inputted information set collecting key is sent to 
the information set obtaining portion 10, and the server 
computer 1 00 then collects the corresponding informa- 
tion sets dynamically from the network or the like using 
this information set collecting key. 
[0102] As described above, the elements of the 
above-described multimedia information arranging ap- 
paratus of the first to fifth embodiments are separately 
provided in the server computer and the client computer, 
thereby constructing the multimedia information arrang- 
ing apparatus of the present invention by the client/ serv- 
er system. 



[0103] The seventh embodiment of the present inven- 
tion is directed to an arranging method allowing a flexi- 
ble arrangement, retrieval narrowing-down and switch- 
ing to another retrieval aspect with respect to an infor- 
mation set using a multimedia information arranging op- 
eration using the above multimedia information arrang- 
ing apparatus described in the first to sixth embodi- 
ments. 

[0104] A conventional retrieval method includes pro- 
viding a retrieval keyword, seeing a retrieval result and 
executing a retrieval again by narrowing down to a fur- 
ther restrictive retrieval keyword, or executing a retrieval 
again by replacing the retrieval keyword with a new one 
when the retrieval result is not the one desired. It can 
be said that the conventional retrieval method executes 
a trial-and-error retrieval by adjusting retrieval key- 
words. 
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[01 05] However, the trial and error is only carried out 
with respect to one feature value as text information, 
which is a retrieval keyword. 

[0106] In the case of carrying out a retrieval using a 
certain feature value of a certain information set as a 
retrieval key, the information set arranging method of the 
seventh embodiment includes, after seeing the retrieval 
result, executing one retrieval after another by using a 
feature value of media information that is different from 
the above media information or by specifying a feature 
value that is different from the above feature value of 
the same media information . In other words, this method 
includes setting an assignment of a feature value ex- 
tracted from each media information to each information 
set arrangement space axis so as to set the information 
set arrangement space having one or more axes, spec- 
ifying a feature value that is different from the one used 
for the arrangement with respect to the arranged infor- 
mation set so as to reset the assignment to the informa- 
tion set arrangement space, rearranging the information 
sets in the information set arrangement space based on 
the resetting, and then setting the axis of the information 
set arrangement space and arranging the information 
sets in the information set arrangement space repeat- 
edly while switching feature values to be used. 
[0107] With this method, after seeing the retrieval re- 
sult, which is an arrangement of the information sets ob- 
tained by a certain feature value, it is possible to try ar- 
ranging of the information sets using a feature value with 
a completely different aspect. Thus, there arises a pos- 
sibility of discovering new information that a searcher 
has never expected. 

[0108] For example, in order to look into the design of 
well-selling women's bags using the multimedia infor- 
mation arranging apparatus described in the first em- 
bodiment, information sets are collected by specifying 
a keyword of "bag," and then arranged in the information 
set arrangement space by assigning a wavelet trans- 
form coeff icient feature value to the X axis, a feature val- 
ue representing the presence of a keyword of woman to 
the Y axis and afeature value representing the presence 
of a keyword of bag to the Z axis as the information set 
arrangement space axes, so that a display result is ob- 
tained. When a searcher discovers an image of a wom- 
an with a white-haired dog carrying a whitish bag among 
the displayed pieces of the image information, for ex- 
ample, the searcher might guess that "a woman going 
out with a dog tends to carry a bag with a color similar 
to the dog." In order to confirm this guess, a feature val- 
ue having a different aspect is assigned to the informa- 
tion set arrangement space axis, thus resetting axis, re- 
arranging and redisplaying the information sets. For ex- 
ample, the combination of a feature value representing 
the presence of the keyword of "bag" and a feature value 
representing the presence of the keyword of "dog" is as- 
signed to the X axis (in other words, the presence of 
both the keywords of bag and dog is specified), a color 
histogram feature value is assigned to the Y axis, thus 



resetting the information set arrangement space and re- 
arranging the information sets, and in this manner, a re- 
trieval with a new aspect becomes possible. It is possi- 
ble to see the obtained result of displaying the arrange- 
5 ment of the image information and use it for judging 
whether or not the guess that a woman going out with 
a dog tends to carry a bag with a color similar to the dog 
is true. 

[0109] Furthermore, in thecase where the information 
10 set is accompanied by other media information, for ex- 
ample, personal information, as text information as de- 
scribed in the first embodiment, when each piece of the 
image information is clicked with a pointing device, a 
new tendency, for example, that a woman taking along 
'5 a bag and a dog with similar colors tends to have a high 
annual income might be discovered. Moreover, in the 
case where audio information of a dog's bark is con- 
tained in the information set as audio information asso- 
ciated with the image information, when a voice frequen- 
ce? cy feature value of the audio information is assigned to 
one of the information set arrangement space axes so 
as to carry out rearrangement and redisplaying, there 
might be a discovery from another aspect. Such a dis- 
covery may be, for example, the tendency that a dog 
25 has a high voice, that is, a dog is small enough to be 
kept inside when many pieces of the image information 
are arranged at coordinates with high voice frequencies. 
In other words, a tendency that "a woman going out with 
a small dog matches the color of a bag with that of her 
30 dog" might be discovered. 

[01 10] As described above, in accordance with the in- 
formation set arranging method of the seventh embod- 
iment, after seeing the retrieval result, which is an ar- 
rangement of the information sets obtained by a certain 
35 feature value, it is possible to try arranging the informa- 
tion sets using a feature value with a completely different 
aspect, thus discovering new information that a search- 
er has never expected. 

40 (Eighth Embodiment) 

[0111] The multimedia information arranging appara- 
tus according to the present invention can be construct- 
ed by computers of several types by recording a pro- 

45 gram, containing the processing operations for realizing 
the operations explained in the above embodiments, on 
a computer-readable recording medium. The recording 
medium, on which the program providing the processing 
operations realizing the multimedia information arrang- 

50 ing apparatus according td the present invention is re- 
corded, can be not only a portable recording medium 
201 such as a CD-ROM 202 or a flexible disk 203, but 
also a recording medium 200 in a recording apparatus 
on the network or a recording medium 205 such as a 

55 hard disk or a RAM in a computer, as illustrated by an 
example of the recording media shown in Fig. 21 . When 
executing the program, the program is loaded into a 
computer 204 and executed in its main memory. 
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Industrial Applicability " 

[0112] According to the multimedia information ar- 
ranging apparatus of the present invention, considering 
a group of related pieces of the media information in- 
cluding the image information, the text information and 
the audio information as an information set, it is possible 
to arrange such information sets in a preset information 
set arrangement space and display the media informa- 
tion according to a viewpoint with respect to this infor- 
mation set arrangement space, 
[0113] Also, according to the multimedia information 
arranging apparatus of the present invention, with the 
information set generating portion, it is possible to col- 
lect pieces of the media information including the image 
information, the text information and the audio informa- 
tion, analyze the relationship between the collected 
pieces of the media information, group together related 
pieces of the media information so as to generate infor- 
mation sets, as a preprocessing before the information 
set arranging operation. 

[0114] According to the multimedia information ar- 
ranging apparatus of the present invention, with the self- 
organizing map processing portion, it is possible to ap- 
ply a self-organizing map operation so as to arrange the 
information sets in the information set arrangement 
space. 

[0115] Furthermore, according to the multimedia in- 
formation arranging apparatus of the present invention, 
with the display viewpoint moving portion, it is possible 
to change the display viewpoint dynamically and display 
the arrangement state of the information sets that a 
searcher wants to see more closely, so as to be dose to 
the display screen surface according to the searcher's 
operation. 

[0116] Moreover, according to the multimedia infor- 
mation arranging apparatus of the present invention, it 
is possible to reset the information set arrangement 
space and carry out a reclassification with a target in- 
formation set being fixed to a specific position, allowing 
a visual recognition of how the arrangement position of 
each information set changes, thus improving conven- 
ience of retrieval and classification operation of the in- 
formation set. 

[0117] In addition, according to the information set ar- 
ranging method of the present invention, after seeing 
the retrieval result, which is an arrangement of the in- 
formation sets obtained by a certain feature value, it is 
possible to try arranging the information sets using a 
feature value with a completely different aspect, thus 
discovering new information that a searcher has never 
expected. 



Claims 

1 . A multimedia information arranging apparatus com- 
prising: 



an information set obtaining portion for obtain- 
ing pieces of media information in units of infor- 
mation sets, which are formed by grouping to- 
gether the pieces of media information that are 

5 related to a same target, the media information 

being of a same kind and different kinds, from 
among a group of pieces of media information 
including image information, text information 
and audio information; 

10 an axis setting portion for assigning an attribute 

of a feature value extracted from the obtained 
pieces of media information contained in the in- 
formation sets to an axis of a space in which a 
group of the information sets is arranged and 

15 setting an information set arrangement space 

with one or more axes; 

a feature value extracting portion for extracting 
a component of the feature value from the piec- 
es of media information in the information sets; 

20 an information set arranging portion for arrang- 

ing the information sets in the information set 
arrangement space based on the attribute of 
the feature value of the pieces of media infor- 
mation contained in the information sets and 

25 the component of this feature value; and 

an information displaying portion for displaying 
pieces of media information corresponding to a 
viewpoint with respect to the information set ar- 
rangement space, from among the pieces of 

30 media information of the information sets ar- 

ranged in the information set arrangement 
space. 

2. The multimedia information arranging apparatus 
35 according to claim 1, wherein, in the axis setting 

portion, a plurality of the attributes of the feature val- 
ues are assigned in combination to one axis of the 
space or one attribute of the feature value is as- 
signed to a plurality of the axes. 

40 

3. The multimedia information arranging apparatus 
according to claim 1 or 2, wherein the axis setting 
portion has an axis resetting function of resetting an 
assignment of the attribute of the feature value to 

45 each of the axes of the information set arrangement 
space and resetting the information set arrange- 
ment space with one or more axes, and 

the feature value extracting portion extracts 
the component of the feature value based on the 
so axis-resetting by the axis setting portion, the infor- 
mation set arranging portion arranges the informa- 
tion sets in the information set arrangement space 
based on the component of the extracted feature 
value, and the information displaying portion dis- 
ss plays the pieces of media information correspond- 
ing to the viewpoint with respect to the reset infor- 
mation set arrangement space. 
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4. The multimedia information arranging apparatus 
according to claim 1 , wherein the information set 
obtaining portion comprises an information collect- 
ing portion for collecting the pieces of media infor- 
mation including the image information, the text in- 
formation and the audio information, a relationship 
analyzing portion for analyzing a relationship be- 
tween the collected pieces of media information and 
an information set generating portion for grouping 
and editing the related pieces of media information, 
which is of the same kind or the different kinds, as 
the information sets. 

5. The multimedia information arranging apparatus 
according to claim 4, wherein the information col- 
lecting portion collects the pieces of media informa- 
tion according to a specified selection criterion 
when collecting the pieces of media information 
from a multimedia information group, and the selec- 
tion criterion is specified by being selected from a 
criterion group including keyword information, site 
information, link information and similarity informa- 
tion with respect to a specific information set. 

6. The multimedia information arranging apparatus 
according to claim 1 , wherein the feature value is 
selected from a DCT coefficient feature value with 
respect to the image information, a wavelet trans- 
form coefficient feature value with respect to the im- 
age information, a HSI color histogram feature val- 
ue with respect to the image information, a feature 
value representing a presence of a specific word 
with respect to the text information, a feature value 
of how many times a specific word is used with re- 
spect to the text information, a voice frequency fea- 
ture value with respect to the audio information, an 
amplitude feature value with respect to the audio 
information and a time change feature value with 
respect to the audio information. 

7. The multimedia information arranging apparatus 
according to claim 1 , wherein the information set ar- 
ranging portion comprises a self-organizing map 
processing portion for utilizing a local interaction 
and performing a self-organization by learning, and 

the information set arranging portion arranges 
the information sets based on the feature value ex- 
tracted by the feature value extracting portion using 
a self -organizing map processing by the self-organ- 
izing map processing portion. 

8. The multimedia information arranging apparatus 
according to claim 1 , wherein the information dis- 
playing portion comprises a display viewpoint mov- 
ing portion having a function of moving a position of 
setting the viewpoint for displaying the information 
set and the information set arrangement space, and 

the information displaying portion displays the 



information set arrangement space, in which the in- 
formation sets are arranged, based on the position 
of the viewpoint set by the display viewpoint moving 
portion. 

5 

9. The multimedia information arranging apparatus 
according to claim 1 , wherein the axis setting por- 
tion resets an assignment of the attribute of the fea- 
ture value to each of the axis of the information set 

10 arrangement space and resets the information set 
arrangement space that has been displayed al- 
ready, 

the information set arranging portion rear- 
ranges the information sets in the reset information 

'5 set arrangement space, and 

when displaying how the information sets are 
rearranged, the information displaying portion 
moves the displayed pieces of media information at 
predetermined intervals from a position at which the 

20 information sets have been located before the rear- 
rangement to a position at which they are to be lo- 
cated thereafter. 

10. The multimedia information arranging apparatus 
25 according to claim 1 or 9, wherein the information 

set arranging portion has 

a function of fixing an information set selected 
by a user to a specific position in an information set 
arrangement space specified by the user, and 

30 a function of fixing the information set select- 

ed by the user to the specific position while rear- 
ranging only the other information sets according to 
the information set arrangement space, when rear- 
ranging the information sets with respect to the in- 

35 formation set arrangement space with reset axis. 

11. A computer-readable recording medium storing a 
processing program for realizing a multimedia infor- 
mation arranging apparatus for arranging and dis- 

40 playing information sets, each of which is a group 
of related pieces of media information including im- 
age information, text information and audio informa- 
tion, in an information set arrangement space, the 
processing program comprising: 

45 

an information set obtaining operation for ob- 
taining the pieces of media information in units 
of the information sets, each formed by associ- 
ating with each other the related pieces of me- 
50 dia information including the image information, 

the text information and the audio information, 
the media information being of a same kind and 
different kinds; 

an axis setting operation for assigning an at- 
55 tribute of a feature value extracted from the ob- 

tained pieces of media information contained in 
the information sets to an axis of a space in 
which a group of the information sets is ar- 
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ranged and setting an information set arrange- 
ment space with one or more axes; 
a feature value extracting operation for extract- 
ing a component of the feature value from the 
pieces of media information in the information s 
sets; 

an information set arranging operation for ar- 
ranging the information sets in the information 
set arrangement space based on the extracted 
component of the feature value; and 10 
an information displaying operation for display- 
ing pieces of media information corresponding 
to a viewpoint with respect to the information 
set arrangement space, from among the pieces 
of media information of the information sets ar- *5 
ranged in the information set arrangement 
space. 

12. An information set arranging method comprising: 

20 

obtaining pieces of media information in units 
of information sets, each formed by associating 
with each other related pieces of media infor- 
mation including image information, text infor- 
mation and audio information, the media infor- 25 
mation being of a same kind and different kinds; 
assigning an attribute of afeature value extract- 
ed from the obtained pieces of media informa- 
tion contained in the information sets to an axis 
of a space in which a group of the information 30 
sets is arranged and setting an information set 
arrangement space with one or more axes; 
extracting a component of the feature value 
from the pieces of media information in the in- 
formation sets; 35 
arranging the information sets in the informa- 
tion set arrangement space based on the at- 
tribute of the feature value of the pieces of me- 
dia information contained in the information 
sets and the component of this feature value; *o 
specifying a feature value that is different from 
the feature value with respect to the arranged 
information sets, resetting an assignment to 
each of the axis of the information set arrange- 
ment space, and rearranging the information 45 
sets in the information set arrangement space 
based on the resetting; and 
setting the axis of the information set arrange- 
ment space and arranging the information sets 
in the information set arrangement space re- so 
peatedly while switching feature values to be 
used. 
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